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Abstract 

Accountability for the preparation of tomorrow’s teachers and administrators has come to 
academia with a full head of steam. Congress has declared that it is the institutions of higher 
education that must be accountable for increasing K- 12 student achievement. As required by the 
1998 U. S. Congress through P. L. 105-244 (2000), colleges of education must begin annual 
submission and reporting of teacher candidate test data. These reports are to be submitted to each 
state department of higher education and to the public. It is anticipated that reporting of test data 
from programs that prepare candidates for educational administration will soon follow (Sewall,. 

A. M., personal communication, May 1 1 , 2000). 

The premise upon which Congress enacted the 1998 Amendments to Higher Education Act 
of 1965 (Title II) was the belief that good teaching is key to student achievement. The Council of 
Chief State School Officers (Leadership for Learning in the 21 sl Century, 2000) reports that 
creating schools to meet public expectations for elementary and secondary school student 
performance will require a new level of leadership to provide the “inspiration, know-how, drive 
and endurance required to change the schools and raise student performance” (p.4). 

The outcome expected to arise from this exercise of accountability is that of individual 
institutions retaining or losing teacher and administrator preparation programs. This paper will 
explore these issues and attempt to create a “report card” of criteria for measuring the quality of an 
educational administration and supervision program. 
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Introduction 

Kleiner (2000) reports parents and politicians are insisting on better teaching in the 
nation’s schools and that change is coming swiftly to schools of education. State politicians are 
responding to this call with legislation mandating that license holders must pass more 
comprehensive and rigorous licensing examinations. National politicians (1998 Amendments, 

2000) are responding by requiring state departments of higher education to report student scores 
on the licensing examinations. 

The research question raised by this proposition and presented in this paper is “How will 
the faculty of an administration and supervision program be able to judge its internal quality 
against external measures of accountability?” The definitions of internal quality (input) and 
external measures (output) by Garrett, Morse, and Samuel (2000) were used in this paper. Internal 
quality referred to attributes the students and faculty brings to the educational experience. External 
measures referred to the interpretation of data used to determine how well the program prepared 
students for success. 

Harvard University, University of Wisconsin at Madison, Stanford University, Teachers 
College at Columbia University, Vanderbilt University, The Ohio State University, University of 
Texas at Austin, Pennsylvania State University, Indiana University, and University of Michigan at 
Ann Arbor are the top ten ranked programs for educational administration and supervision in 2000 
as reported by U. S. News. What determines the quality of these programs? How can other 
programs measure up to the standard set at these universities? Are the evaluation criteria 
appropriate for use by other programs to make improvements? These and other questions have 
been explored in this paper and an outcome of the research has been a report card for programs of 
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educational administration and supervision. The answers to these issues will be explored through 
the concept of program evaluation. 

Program Evaluation 

The term “evaluation” raises the question, “How good is it?” The key question must be 
“Compared to what?” (Robbins, Zweizig, 1988). The Encyclopedia of Educational Administration 
(Dejnpzka & Kapel, 1982) defines program evaluation as, “the assessment of the value, worth, or 
merit of a particular educational program” 

Mohr (1995) explains that program evaluation consists of three primary elements; (a) the 
problem, (b) the outcome of interest, and (c) the subobjectives. Mohr (1995) defined a problem as 
“some predicted condition that will be unsatisfactory without the intervention of the program and 
satisfactory, or at least more acceptable, given the program’s intervention 11 (p. 14): For purposes of 
this research, the problem was to know the standards by which programs of educational 
administration might be judged. Mohr (1995) explains that the outcome of interest is “what the 
program is ostensibly meant to accomplish” (p. 13). The outcome of interest in this study was 
program survival. Mohr (1995) defines a subobjective as “an outcome that must be achieved 
before and in order that some further outcome may be achieved” (p. 31). The subobjective for this 
problem was the identification of the criteria against which a program of educational leadership 
might be judged. These subobjectives become the external measures for determining program 
quality. 

Measures of Internal quality 

One current set of criteria currently used to determine successful graduate school programs 
has been employed by U. S. News (Garrett, Morse, & Flanigan, 2000) to publish a ranking of 
graduate schools that includes programs of educational administration and supervision. The input 
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measures for internal quality used in this ranking system included; (a) program reputation, (b) 
student selectivity (c) faculty resources, and (d) research activity. 

Reputation was used as an evaluative measure by Garrett, Morse, and Flanigan (2000) as 
experts in higher education have long recognized that a university reputation is a valid measure of 
academic quality. Academia has long known that a diploma from a college or university known 
for excellence in a particular field offers graduates a powerful edge in the competition for good 
jobs. Garrett, Morse, & Flanigan (2000) measured reputation of educational administration 
programs by sending surveys to education school deans, associate deans, and a sample of school 
district superintendents in districts of over 5,000 students. The respondents were asked to rate the 
reputation of graduate education programs from “marginal” to “distinguished.” For specialty 
programs such as educational administration and supervision, the deans were asked to identify 10 
schools with the best programs. The superintendents were asked to select the best 25 graduate 
schools of administration and supervision based upon experience in hiring their graduates. The 
reputation criterion accounted for 40 percent of the weighted measure (Education Methodology, 
2000 ). 

Mulrine suggested another factor of reputation involved the status of the faculty and the 
particular department. Alan Goodman, director, of career services at Catholic University, 
suggested perusing the Directory of American Scholars and American Men and Women of 
Science. "They are helpful because they not only provide biographies but also critiques of faculty" 
(Mulrine, p.l). 

Another source of judging reputation is from students in graduate programs. "Find out 
whether students are enthusiastic about the program and whether they feel like they're treated as 
colleagues by faculty," says Jane Finkle, graduate and professional school adviser at the University 
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of Pennsylvania. (Mulrine, 2000, p. 1). Students can also give you a sense of a program's culture. 
"You'd think, wouldn't everyone want to go to a school where there's a nice collegial feeling and 
everyone works together? Not necessarily. You may be the sort of person who thrives in a 
competitive atmosphere and hates group projects” (Mulrine, p. 1). 

Student selectivity in the U. S. News (Garrett, Morse, & Flanigan, 2000) ranking was 
measured by using student test scores (Graduate Record Examination mean verbal, analytical, and 
quantitative), acceptance rates, student-faculty ratio, and number of degrees granted (Education 
Methodology, 2000). The caliber of student that a program attracts influences academic climate 
and attractiveness to graduate programs by high achievers. Student grade point average remains an 
important criterion in determination of quality graduate school programs. 

Faculty resources in the U. S. News ranking (Garrett, Morse, & Flanigan, 2000) involved 
determining ratios of full-time Ph.D. and master’s candidates to full-time faculty. Another 
measure of resources used was the percent of faculty awarded Spencer Foundation-Young Faculty, 
Fulbright, Guggenheim, Humboldt, or American Education Research fellowships in the past two 
years. Also considered were the number of doctoral and master’s degrees granted in the past 
school year and the proportion of the fall students who were Ph.D. candidates (Education 
Methodology, 2000). 

Among the resources important to faculty and graduate students must be that of university 
resources available for research and publishing. Mulrine (2000) writes that Minerva Reed, director 
of career services at Princeton University, says that one of the complaints she hears most often 
from alumni enrolled in graduate schools at other institutions is that the library is insufficient. 

Research Activity in the U. S. News ranking (Garrett, Morse, & Flanigan, 2000) included 
total research expenditures and expenditures per faculty member and the number of faculty 
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members engaged in research. Research expenditures referred to public and private research 
conducted by the school. The Research Activity category accounted for 20 percent of the weighted 
score (Education Methodology, 2000). 

External Measures of Accountability 

There are two significant bodies in most states to which programs of educational 
administration and supervision must comply. Those are the National Council for Accreditation of 
Teacher Education (NCATE) and the State Department of Education for purposes of granting an 
individual the appropriate professional license. 

Kleiner (2000) reports that there are 41 states that now require some type of licensure 
exam. However, most states do not require a licensing exam for new principals and/or 
superintendents (Wargin, 2000). In a few of the states that do require testing, one common 
measure that will be used with increasingly greater frequency to judge program quality and 
candidate performance in many states is the School Leaders Licensure Assessment (SLLA) or the 
School Superintendent Assessment (SLLA) . These assessments are outcomes of the Council of 
Chief State School Officers (20Q0) and the work of the Interstate School Leaders Licensure 
Consortium (ISLLC) to create activities for improving the accreditation of programs that prepare 
education administrators and to measure the skills of tomorrow’s school leaders. 

The SLLA and SSA are products of the Educational Testing Service. The S T I A was first 
used in 1999 by the states of Kentucky, Maryland, Mississippi, Missouri, and North Carolina as 
part of their licensure process for principals. The instrument was developed to provide a thorough, 
fair, and carefully validated assessment for states to use as part of the licensure process for 
principals. It reflects the most current research and professional judgment and experience of 
educators across the country, and is based on both a national job analysis study and a set of 
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standards for school leaders identified by the ISLLC (School Leadership Series, 2000). The SLLA 
has four sections that require open-ended responses. The areas of examination are: (a) Synthesis of 
Information and Problem Solving, (b) Analysis of Information and Decision-Making, 

(c) Evaluation of Actions I, and', (d) Evaluation of Actions II. Each of these sections includes case 
studies or short vignettes describing a school experience. The candidate responds to specific 
questions that may require balancing competing claims for resources, prioritizing action steps, 
articulating the relevant instructional issues, or discussing appropriate problem-solving strategies 
(Wargin, 2000). 

The National Council for Accreditation of Teacher Education (2000) is the accrediting 
agency for many programs of educational administration and supervision. The Executive Board 
adopted the NCATE 2000 Unit Standards on May 11, 2000. Accountability for meeting these 
standards will be required for all accreditation visits beginning in fall 2001. All six of the NCATE 
standards will influence the selection of criteria used for accountability of administrator 
preparation programs. Standard 1 (Candidate Knowledge, Skills, and Dispositions) requires the 
institution to prepare candidates to meet professional, state, and institutional standards. Standard 2 
(Assessment of System and Unit Evaluation) mandates that the program collects and analyzes data 
on applicant qualifications and performance. Standard 3 (Field Experiences and Clinical Practice) 
requires the program to work with school personnel to develop and demonstrate the knowledge, 
skills, and dispositions necessary to help all students learn. Standard 4 (Diversity) requires that 
candidates work with a diversity of P-12 students and higher education faculty. Standard 5 
(Faculty Qualifications, Performance, and Development) require that faculty aie qualified and 
model best professional practices in scholarship, service, and teaching. Standard 6 (Unit 
Governance and Resources) implies that the unit has the leadership, authority, budget, personnel, 
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facilities, and resources for the preparation of candidates (National Council for Accreditation of 
Teacher Education, 2000). 

Wise and Golnick (2000) report that NCATE-accredited colleges will be expected to assess 
themselves for the effectiveness of their programs based on candidate performance. A part of the 
institutional assessment system would provide comprehensive information on candidate 
performance to include content knowledge, how to teach content effectively, and effect on student 
learning. 

Improving Input Measures of Program Quality 

Reputation is defined as “what people think and say the character of a person or thing is; 
without disgrace or discredit” (World Book Dictionary, 1963). The Harris-Fombrun Reputation 
Quotient (2000) is used as a business tool to rate a company’s rating of reputation among its 
competitors. The scale comprises six dimensions of reputation; five of appear to be transferable to 
examining reputation of a university program. The dimensions and their attributes are as follows: 

o Emotional Appeal (the organization offers a good feeling, admiration and respect, and 
trust) 

© Products and Services (the organization stands behind its products and services, is 
innovative, offers high quality, and a good value for the money) 

© Vision and Leadership (the organization has excellent leadership and a clear vision of its 
future). 

o Social Responsibility (the organization supports good causes and maintains high standards 
in the way it treats people) 

Student selectivity is closely related to reputation in that students, when possible, choose 
graduate schools that will enhance their own marketability. As it is said “success breeds success,” 
high achieving students choose to associate themselves with that success. As the greatest majority 
of students do not have the luxury of full-time studies, students tend to make a selection of 
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graduate school based more for region and availability than for national reputation. McCarthy 
(1999) reported that a 1978 study by Davis and Spuck found that the majority of master’s students 
and two-fifths of doctoral students lived within 25 miles of the university they were attending. 

This selectivity factor then becomes important for programs to attract the most capable of regional 
candidates. 

Influencing some of these student selectivity factors could be achieved only through 
controversial actions involving tougher admission standards, reduction of emphasis on teaching by 
the professorate and better preparing students for the examinations (Garrett, Morse, & Flanigan, 
2000 ). 

Standardized test scores are a common measure of selectivity for the top-ranked graduate 
schools. A wide range of average GRE scores was reported by Garrett, Morse, & Flanigan (2000). 
In comparing GRE scores of the highest ranked educational administration programs on the U. S. 
News: Graduate Schools, Administration/Supervision, the highest average scores seem to be by 
regions. Taking an average of the top four schools in each region, the reported GRE scores ranked 
in the following order: west (1815), central mid-west (1717), east (1708), north (1692), and south 
(1665). Better preparing students for entrance examinations may well be one method of raising 
these scores. The significant issue to be considered in taking this action would be that of 
confirming whether or not these scores actually contribute to graduates that will make significant 
changes to society and the body of knowledge. 

Faculty resources are a significant area of concern for the improvement of educational 
leadership programs (Achilles, 1999; Misltein, 1999; McCarthy, 1999). One concern raised in 
many states has been the large number of programs that prepare educational administrators. By 
economy of scale, it is only reasonable that with several preparation programs in a state, many 
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institutions have a faculty body in administrator preparation of one or two members. Milstein 
(1999) maintains that quality preparation cannot be maintained when so few faculty are 
responsible for all program design and delivery activities. 

The related problem of too many programs is that of a reduction in faculty availability in 
educational leadership (Milstein, 1999). With fewer faculty available for open positions, many 
programs remain understaffed. Not having the full compliment of faculty increases faculty-student 
ratios and decreases program quality. 

Research activity , or lack thereof, has been a concern for many in the field (Achilles, 1999; 
Milstein, 1999; McCarthy, 1999). Achilles (1999) writes that educational leadership faculty have 
not fulfilled their responsibility for conducting quality research of their own field of study. 

Based upon the changing demographics of the educational leadership faculty, some causes 
for lack of productive research may include; (a) time is not available because of the large number 
of students in programs with so few faculty members; (b) few faculty members in a program 
creating greater teaching and advising loads; (c) the “graying” of the professorate may lead to less 
research productivity as faculty look to retirement; (d) fewer faculty from which to choose as 
vacancies occur; (e) the shift from a predominantly male field in 1974, 98%, (McCarthy, 1999) to 
20% female by 1994, and; (f) the growing number of faculty members who have been in the field 
as educational administrators. It may be too early to judge the impact of these changes on research 
activity, but they may become important future considerations. 

In Conclusion 

Accountability in education is not a new subject. Accountability for outcomes is a new 
subject for the field of educational administration preparation programs. As state legislatures and 
Congress have begun to place a significant burden on institutions of higher education for 
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improving the quality of teachers and administrators in the field. As greater scrutiny is given 
programs in higher education, the stakes of program survival become a more severe reality. It may 
soon come to pass that university preparation programs that do not graduate students that are able 
to clearly demonstrate what they know and how they can successfully do it will surely fade away. 
The wave of the future may well be an addition to the “publish or perish” adage to that of 
“produce and publish or perish.” 
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